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METHOD AND SYSTEM FOR TRACKING SCREEN ACTIVITIESY FOR MULT I- 
SCREEN TRANSACT I ONS 

rooon CROSS reference to related application 

10 This is a continuation-ifHpaft of co - pend i ng U.S. application No.: 

09/ 4 92,559, entit le d "M e thod and Syst e m for Track i ng Scr ee n Act i vit ie s i n 
On li nQ Transact i onc", fi l ed on January 27, 2000, and oo p e nd i ng U.S. 
app l icat i on No.: 09/6 44 , 4 0 4 , ent i t l ed "Method and System for Tracking 
Scre e n Activity fi l ed on August 23, 2000, a ll by the applicants thereof 

15 09/705.391. now US Pat. No.: 6.662.226 . 

r00021 BACKGROUND OF THE INVENTION 
roOOSI Field of the Invention 

[00041 The present invention relates to interactions with terminal 

devices having a user interface to display information, and more particularly to a 
20 method and system for capturing, tracking, storing, augmenting and replaying the 
displayed information encountered during one or more interactive session, wherein 
the one or more interactive session involves a series of screen displays. 

[00051 Description of the Related Art 

[00061 ^Various infomiation is accessed through wide area networks 

25 (WAN) such as the Internet. The content of these wide area networks is dynamic, 
infomiation is always being added and deleted. Infomiation available at one time 
may be unavailable at a later time because a user may not save it when it was 
available or its source may have been deleted, or it may be still available but the 
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5 user is not quite sure of its location. In other cases, one desires to see how others 
proceed with interactions with provided information. 



information streams such as interaction with various web sites over the Internet. 
These interactions are typically achieved via hardware devices including a display 

10 device and a user interface (i.e., a keyboard and a pointing device). In order to 
instruct a computing system to perform a task, a user may type a command on a 
keyboard or make a selection from a menu or button bar using a pointing device 
(e.g.. a mouse or touch pad). In response to the user's input, the computing 
system may display text on the display device, display an image or play a sound, 

15 all of which provides an indication to the users of the results of their interaction. 

fOOOSI Selectively preserving some of these interactions is akin to 

transmitting the information associated with a particular interaction from the 
present to the future. There is therefore a need for information retention, source 
identification, and processing services associated with the transmitted information 
20 from designated terminal devices. Additionally, since this information must be 
stored for future access, there is also a need for information annotation (i.e.. from 
where, for who and when). 



r00071 



-Computing devices enable users to interact with various 
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SUMMARY OF THE INVENTION 
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An object of the present invention is to provide a method 



and system that facilitates an individual, enterprise or business entity to 
record, store, process and manage information associated with 



interactions transacted on designated terminal devices. More 
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5 specifically, individual and sequenced display presentations are 

captured or recorded and supplemented with associated infomnation 
such as input control sequences, time stamps, user specific 
identification information, processing instructions, source information 
and validation infonmation which may take the form of alpha numeric 

10 sequences or machine readable entities either of which may be 

encrypted, 

rOOHI in an embodiment of the present invention, software 

agents (resident on a user's terminal device and/or a remotely 
accessible server devices) record the activities associated with a given 

15 Interactive session where that interactive session comprises a series of 

screen displays and any associated user interface or network activity. 
The captured information is indexed, processed (i.e.. sensitive 
information may be encrypted and the captured content may be 
compressed for efficient storage) and stored for future access (i.e., 

20 replay or playback) either on the initiating terminal device or a 

designated remote server device. Additionally, since source addressing 
information is retained with the captured information, future 
modifications to source files associated with the captured information 
may also be tracked either offline or upon request. 

25 f 001 21 The present invention can be implemented in numerous 

forms. Different implementations of the present invention yield one or 
more of the following advantages. One advantage of the invention is that 
an effective digital tracking system is provided to track device 
interactions (I.e., screen activity over a period of time and associated 

30 input and network interactions) associated with user interactions with 
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various information streams accessible through a network such as the 
Internet or an Intranet. This capturing process may be activated 
automatically or may be initiated by the user as desired. Another 
advantage of the present invention is that the captured information can 
be processed (i.e., encryption, compression) augmented by time 
stamps, user specific infomnation (i.e., electronic signatures) and 
validation information which may take the form of alpha-numeric 
character strings or machine readable entities which may be encrypted. 
Still another advantage of the present invention is the captured 
Information may be stored and managed on initiating terminal device 
and/or remotely on a designated server device for future reference. 

f00131 — The foregoing and other objects, features and 

advantages of the invention will become more apparent from the 
following detailed description of a preferred embodiment, which 
proceeds with reference to the accompanying drawings. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



r00151 — The present invention will be readily understood by the 

following detailed description in conjunction with the accompanying 
drawings, wherein like reference numerals designate like structural 
elements, and in which: 

10 [00161 Figure 1 is a block diagram of a networked communication 

system which may be used to implement the method and system 
embodying the invention; 

r00171 Figures 2 shows a functional diagram of a TSR 

(Transaction Recording System) Server which may be used in 
15 conjunction with the implementation of an embodiment of the present 

invention; 

[00181 Figures 3A to 3G illustrate representative screen displays 

which may be captured, validated and processed in accordance with an 
embodiment of the present invention; 

20 [00191 Figure 4 illustrates a representative series of screen 

displays which may be captured in accordance with an embodiment of 
the present invention; 

[00201 Figure 5 is a flow diagram of the process associated with 

processing (i.e., encryption, validation and compression) a series of 
25 captured screen displays in accordance with an embodiment of the 

present invention; 
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[00211 Figure 6A is a process of a terminal device being 

interacted with a user; 

f00221 Figure 6B is a process of a server communicating with the 

tenninal device in Figure 6A; and 

r00231 Figure 6C shows a process of retrieving the archived file 

at a terminal associated with the server of Figure 6B. 

r00241 DETAILED DESCRIPTION OF THE INVENTION 

f00251 ^The invention pertains to a method and a system for 

capturing, processing and replaying a series of screen displays 
associated with one or more transactions. As used herein, a transaction 
means data exchange between two devices. For example, a transaction 
occurs when a provider accepts an order of certain items from a buyer. 
Likewise a transaction is deemed to occur when pertinent 
data/information entered/provided by one side is released to the other 
side. In general a transaction means a sequence of information 
exchange and related work (such as database updating) that is treated 
as a unit for the purposes of satisfying a request and for ensuring 
database integrity. For a transaction to be completed and database 
changes to occur, a transaction has to be completed in its entirety. One 
of the common transactions over the Internet is a catalog merchandise 
order by a customer. When the order is received on the server side, the 
order transaction involves checking an inventory database, confirming 
that the item is available, placing the order, and confirming that the order 
has been placed and the expected time of shipment. If the order is 
deemed as a single transaction, then all of the steps must be completed 



and validated before the transaction is successful and the database is 
actually changed to reflect the new order. 

[00261 Generally, a transaction is supported by one or more 

screen displays so that a user can interact with the displays to conduct 
the transaction. One of the key features in the present invention is to 
capture such displays including user's interaction. Another feature is to 
provide a mechanism to store and manage the captured displays. Still 
another feature is to playback the stored displays so that review of the 
displays reveals how a user interacts with his/her terminal to proceed 
with a transaction. In one embodiment, the stored information as a 
series of captured screen interactive infomnation (i.e., screen displays 
and user inputs) can be augmented with validation information and 
played back in a predefined order upon request. 

r00271 Terminal devices, also refen^ed to as computing devices 

herein, include but are not limited to personal computers, laptop 
computers, computer terminals, personal digital assistants, palm-sized 
computing devices, cellular phones, two-way pagers and computer work 
stations. Such devices typically have a user interface comprised of a 
display, a keyboard/keypad and a pointing device (e.g.. a mouse, a 
trackball, a joystick, a navigation key-set or a touch-pad). 

f00281 The detailed description of the invention is presented 

largely in terms of procedures, steps, logic blocks, processing, and other 
symbolic representations that directly or indirectly resemble the 
operations of data processing devices coupled to networks. These 
process descriptions and representations are typically used by those 
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skilled in the art to most effectively convey the substance of their work to 
others skilled in the art. Reference herein to "one embodimenf or "an 
embodiment" means that a particular feature, structure, or characteristic 
described in connection with the embodiment can be included in at least 
one embodiment of the invention. The appearances of the phrase "in 
one embodiment" in various places in the specification are not 
necessarily all referring to the same embodiment, nor are separate or 
alternative embodiments mutually exclusive of other embodiments. 
Further, the order of blocks in process flowcharts or diagrams 
representing one or more embodiments of the invention do not 
inherently indicate any particular order nor imply any limitations in the 
invention. 

f00291 Refen'ing now to the drawings, in which like numerals refer 

to like parts throughout the several views. Figure 1 shows a basic 
system configuration in which the present invention may be 
implemented in accordance with a preferred embodiment. 
Communications system 100 generally includes a plurality of 
communications networks such as wireless network 102 with an 
associated SMS channel and data network 104 (i.e., the Internet or a 
private intranet). These communications networks support 
communications between a plurality of diverse terminal devices, such as 
is illustrated by wireless communication device 108 (i.e.. a cell phone), 
personal digital assistant (PDA) 112, personal computer 124 and a 
plurality of network servers represented by network server 150. 



r00301 ^According to one embodiment of the present invention, 

services associated with the capture and storage of a terminal device 



activities are processed by software modules resident on the initiating 
terminal device (i.e., PDA 112) or on remote server devices such as 
Transaction Recording System (TRS) Server 140. These software 
modules provide services relating to the acquisition, processing (i.e.. 
compression and encryption), validation and management of screen 
display content (I.e., a series of screen displays), input commands, 
device status information, user specific infomiation, source specific 
information, network information, resource utilization (i.e., printers and 
external storage devices), time stamps and in general any information 
associated with the user's interaction with the terminal device. 
Additionally, the software modules provide services related to the 
management and utilization of local and remote user specific storage 
areas. 

rOOSn The communication system illustrated in Figure 1 is 

provided for purposes of illustration and not limitation. It would be 
understood by one of ordinary skill in the art that the present invention 
may be practiced in a communications system having configurations 
and system components that differ from those described above. 

r00321 ^According to one aspect of the present invention, a user 

interacting with a terminal device (i.e., PDA 112), via an associated user 
interface (i.e., a touch screen and associated display), may initiate the 
capture process to capture a series of displays on a screen through a 
pre-defined user interface interaction (i.e., a soft key on PDA 112). 
Upon process initiation, software modules resident on the terminal 
device or embedded as an applet or an application cause a series of 
screen displays and associated device and network interactive event 
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indications to be captured. The captured images and event indications 
may be supplemented with validation information (i.e., a time stamp, a 
user specific electronic signature and validation Information which may 
take the form of alpha numeric character strings or machine readable 
marks) and then processed for storage and future display. The archived 
event information may be stored locally and/or on a designated remote 
server device for future reference. 

r00331 It is defined without the loss of generality that an 

interaction involves activities (i.e., display content, user input/output in 
response to what Is being displayed in the subject terminal device) 
performed by a user with respect to the user interface of a terminal 
device that may or may not be networked. Some key advantages of the 
present invention include the ability to archive device interactions, 
validate those interactions (i.e., an encrypted alpha numeric character 
string which encodes a portion of the archived content) and associate 
the archived information with attribute information such as electronic 
signatures or time stamps. 

r00341 Referring now to Figure 2, there is shown a functional 

block diagram of a TRS server 240 that may conrespond to TRS server 
140 of Figure 1 . A network interface 242 in TRS server 240 facilitates a 
data flow between a data network (i.e., data network 104 of Figure 1) 
and TRS server 240 and typically executes a special set of rules (a 
protocol) for the end points in a link to send data back and forth. One of 
the common protocols is TCP/IP (Transmission Control Protocol/Internet 
Protocol) commonly used in the Internet. The networi^ interface 
manages the assembling of a message or file into smaller packets that 
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are transmitted over the associated data network and reassembles 
received packets into the original message or file. In addition. TRS 
server 240 handles the address part of each packet so that it gets to the 
right destination. 

r00351 TRS server 240 comprises a processor (or multi- 
processor) 248, a server module 242 and a storage space 246. In 
practice, any computing device having reasonable computing resources 
(i.e.. processing power and memory capacity) may be utilized as an 
TRS server. Storage space 246 may be resident within TRS server 240 
or in a separate accessible server device (not shown). Part of the 
storage space 246 is allocated to retain captured infomnation from the 
client devices and accessible upon request. It should be noted the 
storage space 246 may be a single storage device or a cluster of 
storage devices located locally and/or remotely (e.g. storage space 249 
is connected through a network). In one embodiment, the captured 
information may be respectively associated with a particular user, type 
information of the client device, billing infomfiation, electronic signatures, 
device infomiation etc.. 

r00361 ^According to one embodiment of the present invention, 

server module 242 is a compiled and linked version of a computer 
language implementing the present embodiment and loaded in a 
memory. When executed by TRS server 240 (i.e. processor 248), server 
module 242 performs a number of functions to facilitate the operations 
associated with a preferred embodiment of the present invention. 
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r00371 Server module 242 comprises a management module 

242a, validation module 242b, attribute module 242c, security module 
242d, indexing module 242e and content analysis module 242f. 
Management module 242a provides account initialization, management 
and service functions for a plurality of user accounts associated with 
users having access to this service. In one embodiment, management 
module 242a is an interface selectively accessible by users and an 
administrator respectively. Typically, a user is permitted to retrieve the 
recording of his/her screen activities while an administrator is permitted 
to retrieve any one's recording. 

r00381 ^Validation module 242b provides validation services that 

serve to generate evidence that the archived event information has not 
been modified. The evidence may take the form of alphanumeric 
character strings containing encoded or encrypted infomnation 
associated with the archival information. For example, a portion of the 
archived event information may be incorporated into a machine- 
readable symbol or a certified copy may be accessed via a trusted third 
party server device. Attribute module 242c associates additional 
information (i.e., time stamps, electronic signatures, position information, 
etc.) with the archived event information. Security module 242d protects 
the archived information from unauthorized access. Indexing/Routing 
module 242e provides services relating to cataloging and providing an 
index of available event files. Content analysis module 242f is used to 
analyze what is in the captured events (i.e. screens). In one 
embodiment, content analysis module 242f employs an OCR engine to 
perform optical character recognition of one or more selected pages of 
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the captured events. Together with indexing module 242 , content 
analysis module 242f may help automatic indexing of the captured 
events. It is clear to those skilled in the art that not every element in the 
module 242 needs to be implemented to achieve the desired results 
contemplated by the present invention. 

r00391 Figures 3A to 3G illustrate a series of representative 

screen displays that may be captured, validated and processed in 
accordance with an embodiment of the present invention. The screen 
displays may be displayed on a display screen of a computing device, 
such as a personal laptop/desktop computer and a personal data 
assistant (PDA). 

r00401 ^As used herein, a display screen or a screen is the 

physical display apparatus in a computing device, such as a 15 inch 
CRT or LCD screen commonly seen with a computing device. A screen 
display, a displayed page, a displayed window or simply a display is an 
image presented on the display screen. For example, a file that 
constitutes a display may be an HTML file, wherein HTML stands for 
HyperText Markup Language, an image thereof appears on a display 
screen when the file is read or executed by a display application. 

[00411 ^To understand the representative screen displays in 

Figures 3A to 3G, a PDA is used as one of the exemplary computing 
devices. PDA 312 may correspond to PDA 112 of Figure 1 and includes 
a user interface comprised of a display/touch screen, a navigation key- 
set, application keys and softkeys which may be used to interact with 
the PDA 312. In accordance with an embodiment of the present 
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invention, the TRS archiving can be initiated through a TRS program 
key 315, a touch of a designated area on a screen display, an entry of a 
predefined word or web address. 

f00421 In one embodiment, the TRS archiving is initiated when a 

user logs onto a particular screen display, such as an initial page of a 
personal account. Once TRS archiving is initialized, user interactions 
with PDA 312 (i.e. screen content, user inputs, URI's, and associated 
cached information) for the series of screen displays illustrated in 
Figures 3A to 3G are sequentially captured. 

f00431 Each of the screen display is captured by a screen 

capturing process that may be an application or a Java applet 
embedded in a display. In one embodiment, the screen capturing 
process is implemented by utilizing calls provided in Microsoft 
Foundation Class (MFC) supported in MS Windows OS. An example of 
such implementation is provided as follows: 

BOOL CaptureCurrentScrollArea(int nHeight) 
{ 

// create a Xlmage object 

// Xlmage is a predefined class and contains image 
// attributes and device dependent bitmap object. 

Xlmage* pimage = new Xlmage; 

// assign the image attributes. 

// xJWidth is the image width being captured. 

// nHeight is the height for the scroll area. 

// g_uSysBitCount is system graphics bit/pixel setting. 

plmage->xJWidth = xJWidth; 
plmage->xJHeight= nHeight; 
plmage->x_wBitsPixel = g_uSysBitCount; 

// create screen DC 
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// this DC will be used to copy bitmap from the scroll area 
CDC dcScr; 

dcScr.CreateDCC'DISPLAY", NULL. NULL, NULL); 

// create a device dependent bitmap object. 

if (!plmage->bitmap.CreateCompatibleBitmap 
(&dcScr, plmage->xJWidth, plmage->xJHeight)) 
return FALSE; // the capture is not successful. 

// an empty bitmap is created. 

// now create a compatible memory DC. 

CDC dcMem; 
BOOL bCreateDC = 

dcMem.CreateCompatibleDC(&dcScr); 
CBitmap* pOldBmp = 

dcMem.SelectObject(&plmage->bitmap); 

// ready to copy the bitmap from the scroll area. 
// BitBIt is the critical function call to do the job, 
// g_rcCapture is the global variable that contains the 
// coordinates of the current window being captured. 

dcMem.BitBlt(0, 
0. 

plmage->xJWidth, 

plmage->x_IHeight, 

&dcScr, 

g_rcCapture.left, 
g_rcCapture.bottom - nHeight, 
SRCCOPY); 
dcMem.SelectObject(pOldBmp); 

// the job is done. 

// the Xlmage object contains the captured bitmap of 
//the scroll area. 

// store the captured image in a local file. 

if (!SaveBitmapToTempFile{plmage)) 

return FALSE; // the capture is not successful. 

// calculate the total height of the complete image 

g_nTotalHeight += nHeight; 
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II the saved image file will be used in the image 
// concatenation when we stitch individule pieces 
// of the document image back into its entirety. 

return TRUE; // the capture is successful. 

} 

wherein g_rcCapture is the global variable that contains the coordinates of 
the active window being captured. The only parameter passed by calling 
module is the height of the display window that can be readily determined. 

f00441 Once the displays in Figures 3A to 3G are captured, the 

displays or contents therein may be supplemented with attributes such 
as a time stamp to indicate when the displays or contents are captured. 
The displays or contents therein may be also processed such that the 
displays selective or content elements can receive special treatment 
(e.g. encrypted) and afterwards be compressed using a compression 
process. In a preferable embodiment, the displays or processed 
versions thereof are uploaded to a remote server through a network 
(e.g. a LAN, a wireless network, or the Internet). 

[00451 Figure 4 illustrates a representative series of screen 

displays that have been captured in accordance with screen displays 
shown in Figures 3A to 3G. Validation information for the series of 
screen displays may be associated with each individual screen display 
or may be grouped into a single validation entity that represents the 
group as a whole. Additionally, there may be separate validation entries 
for each of the individual screen displays and the series of screen 
displays as a whole. As shown in Figure 4, each of the captured 
displays has been respectively attributed with a time stamp to indicate 
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when the display is captured at the terminal device or anrived at the 
server, 

[00461 Figur 5 is a functional diagram of a process associated 

with processing (i.e., encryption, validation and compression) a series of 
captured screen displays 501 in accordance with an embodiment of the 
present invention. Each of screen displays 501 is initially cached at 501 
and may be processed (e.g. texts are recognized by an OCR process) 
or parsed so that content elements requiring special treatment may be 
proceeded. For example, some of the content elements contains 
classification tags (i.e., "classified" and "unclassified") then those 
elements (i.e., credit card numbers, account numbers etc.) can be 
singled out for special treatment such as encryption. At 505 validation 
data such as time stamps or electronic signatures are added and the 
screen displays (including the attributed ones) may then be 
compressed. The processed file is stored in a TRS database at 507 that 
may be resident on the subject terminal device or a remote server 
device. Upon request, the stored file can be retrieved and replayed at 
509. In one application, a viewer can assess what transactions a user 
(i.e. a person who interacts with the terminal device) has entered and 
how the user was reacted to the transactions. The assessment may 
potentially provide useful marketing feedback. 

[00471 Figure 6A shows a process 600 of a terminal device being 

interacted with a user. Generally, the terminal device is coupled to a 
server and provides a mechanism to the user to perform certain 
functions. For example, the server is a brokerage (e.g. 
www.schwab.com ) from which the user can trade stocks during 
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business hours or the server is an internal data center with which the 
user can exchange data through the terminal device. After the user 
logins onto the server and navigates to a page or display that may be 
the initial display for a transaction. According to one embodiment, when 
the initial display embedded with a flag or a signal that can trigger the 
TRS process comes in at 602, a screen capturing application is 
activated at 604. As described above, the screen capturing application 
may be preinstalled in the terminal device or downloaded with or 
embedded in the initial display. 

r00481 ^As the user interacts with the initial display or proceeds 

with subsequent displays, the displays are automatically captured by the 
screen capturing application at 606. The user's interactions with the 
tenninal device may include an entry of data, one or more 
words/phrases, a click of a designated area in the display. As an option 
at 608, the captured displays may be attributed with one or more notions 
that include a timestamp, the user's identity infonmation or device 
information of the terminal device. 

r00491 ^At 610, it is determined if the respectively captured 

displays are to be cached. Depending on an exact implementation, each 
of the captured displays is respectively transported to a storage device 
at 612, hence there is no need for caching a captured display. The 
storage device may be a server located separately with respect to the 
terminal device and coupled thereto through a data link under a session 
(e.g. opened for the transaction). Whenever a display is captured, the 
image thereof is transported to the storage device. At 614, the process 
600 goes to 606 to capture another display till no more displays need to 
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be captured, which can be indicated within a last display to inactivate 
the screen capturing application. When no more displays need to be 
captured, the process 600 ends. 

roOSOl Depending on another implementation that required that all 

captured displayed are transported together to the server, then a 
captured display is to be cached at 610, and the captured display is 
stored in a memory space 616. At 618, the process 600 may go to 606 
to capture another display till no more displays need to be captured, 
which can be indicated within a last display to inactivate the screen 
capturing application. At 620, the captured displays are then transported 
to the server and the process 600 end. 

r0051l It should be noted that an optional procedure may be 

provided just before 612 or 620 to process the captured display(s). In 
one embodiment, the captured displays in image format can be 
compressed using a commonly known image compression technique 
such as JPEG. In another embodiment, the captured images are 
compressed using a commonly known image compression technique 
such as MPEG or other compression to take the advantage of similarity 
between two immediate displays. The purpose is to reduce the data size 
so that the transportation of the captured displays to the server is more 
efficient over the data line. In any event, the captured image or images, 
regardless processed or not, are referred to as a file that may be 
resident locally or being transported to the server. 

r00521 Figure 6B shows a process 640 of a server 

communicating with the terminal device as discussed in Figure 6A. At 
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642, the server awaits a request if any file is upcoming from a terminal 
device such as the one in Figure 6A. When a request is received, the 
server assigns an identification to the file so that a query can be made 
later to retrieve the file when there is a need to review the file. 
Depending on an exact implementation or application, the identification 
may be a session ID, a transaction ID or any ID that can uniquely 
identify the file. At 646, the file is received. At 648, it is determined if 
there is another file related to the arrived file. As described above, 
sometimes, there are multiple files each comes individually for one 
transaction and sometime there is only one compounded file. If it is 
determined that there are no more files, the anrived files are then kept in 
a storage space at 650 for future retrieval. 

f00S31 Figure 6C shows a flow process 660 of retrieving the 

archived file at a terminal associated with the server. At 662, the 
process 660 awaits a retrieval or replay request that may come from a 
terminal operated by an operator. The request may include parameters 
related to a transaction ID or user identifier infomnation so that a correct 
file may be located. For example, a business desires to review a user's 
behavior with respect to its web site, the process 600 can be used to 
view how the user interacts with the web site. A request may include a 
query of the user's ID. When the replay request is received, the process 
660 is preferably instructed how to play back the archived file, namely in 
a specified order. At 666, the process 660 proceeds to look up for the 
file. At 668, the file is played back and displayed on a screen for review. 

[00541 ^The invention may be implemented as a method, a system 

or code on a computer readable medium (i.e. a substrate). The 
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computer readable medium is a data storage device that can store data, 
which can thereafter, be read by a computer system. Examples of a 
computer readable medium include read-only memory, random-access 
memory, CD-ROMs, magnetic tape, optical data storage devices and 
canier waves. The computer readable medium can be distributed over 
a network coupled computer system so that the computer readable code 
is stored and executed in a distributed fashion. 

rOOSSI The advantages of the invention are numerous. Different 

embodiments or implementations may yield one or more of the following 
advantages. One advantage of the invention is that user's interactions 
with terminal devices can be archived in a form that could be hardly 
altered (e.g. an image of a screen display) and validated for future use. 
Another advantage of the invention is that supplemental information may 
be associated with the archived content (i.e., time stamps, electronic 
signatures, location information, subject, and indexing). Still another 
advantage of the invention is that the individual screen displays may be 
stored as a single file that can be played back to show how each of the 
screen displays is formed. 

f00S61 ^The many features and advantages of the present 

invention are apparent from the written description, and thus, it is 
intended by the appended claims to cover all such features and 
advantages of the invention. Further, since numerous modifications and 
changes will readily occur to those skilled in the art, it is not desired to 
limit the invention to the exact construction and operation as illustrated 
and described. Hence, all suitable modifications and equivalents may 
be considered to fall within the scope of the invention. 



22 



5 



12/06/03 




■3- 




12/06/03 



Claims 

We claim: 

4: A m e thod for archiv i ng an i ntoraot i on by a us e r with a t e rm i na l dev i oo 

hav i ng a d i sp l ay m e ans and a random acc e ss memory, the method 

compris i ng: 

re a d i ng a ser i es of user i nt e rfac e e v e nts from th e random a cc e ss 

memory i n the termina l d e v i c e ; wh e r e in th e s e r ie s of user i nterfac e o v o nts 
r e pr e s e nts a series of scr e en d i splays be i ng displayed during a per i od of 
t i m e ind i cated by a n i n i t i at i on s i gnal and a t e rm i nat i on s i gnal; 

g e n e rat i ng on e or mor e f ile s r e pr e s e nting th e s e r ie s of us e r i nt e rface 

e vents; and 

stor i ng th e g e n e rat e d on e or mor e f i l e s in a storag e ar e a. 

Sr. The method as recited i n c l aim 1 further comprising: g e n e r a t i ng on e 

or more attr i but e s to b e as s oc i ated with e ach of th e on e or mor e fi l es. 

^. Th e m e thod as r e cited i n cla i m 2, wh e r ei n the one or mor e attr i butes 

i nc l ud e s an a l phanum e r i c charact e r str i ng. 

4: Th e method as r e c i t e d i n cla i m 3, wher ei n th e al phanum e r i c 

charact e r str i ng i s e ncrypt e d. 

Th e m e thod as r e cit e d i n cla i m 1 further compr i s i ng: generat i ng on e 

or mor e attr i butes to be associated w i th each of user interface e vents. 
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The m e thod a s r e c i t e d i n claim 5, wh e re i n th e on e or moro attr i but e s 

inc l ude one or mor e of (1) a t i me stamp, (2) an o l ootron i o signatur e . (3) 
t e rm i na l dev i ce l ocat i on i nfonmation. ( 4 ) informat i on r el at i ng to the t e rm i n al 
d e vic e . (5) us e r i nformation, (6) r e lat i v e s e quence i ndex and (7) system 
provided i nformat i on. 

7-, Th e m e thod as r e c i t e d i n c l a i m 1 , wh e r ei n e ach of th e scre e n 

d i splays i ncludes an i nteract i on of the user with the t e rminal devic e . 

8.^ Th e m e thod as r e cit e d in cla i m 7, wh e r e in the i nt e ract i on i nclud e s 

one or moro of (i) an entry by th e us e r, (i i ) a c li ck by the us e r and ( i i i ) a word 
or phras e . 

ft Th e m e thod as r e c i ted i n claim 1 . wh e r ei n th e g e n e rat i ng of th e on e 

or more fil e s Includes compr e ssing th e f i l e accord i ng to a compression 
sch e m e . 

4ft Tho method as r e cit e d i n cl ai m 1 , where i n each of th e us e r i nt e rfac e 

events is an image. 

Th e m e thod as r e c i t e d i n cla i m 10. wh e re i n each of the on e or mor e 

fi le is a n i mage r e pr ese ntat i on of th e us e r i nt e rfac e e v e nts. 

4Sr. Th e m e thod as r e cit e d in c l a i m 1 , wh e r e in th e g e n e rating of th e on e 

or moro fil e s i nc l ud e s sele ct i v e ly proc e ss i ng e le m e nts contain e d with i n 
each of the one or more f ile s. 
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4^. — Tho method as r e cit e d in c l a i m 12, wh e r ei n th e s e l e ctiv el y proc e ssed 
ele m e nts are e ncrypt e d. 

44^ — Tho m e thod as rec i t e d in cla i m 1 , wh e re i n th e storage area is l ocat e d 
r e mot el y and acc e ss i bl e through a data l ink. 

4§ Th e m e thod as r e c i t e d i n c l aim 1 4 , wh e r ei n th e stor i ng of th e 

g e n e rat e d on e or more f i les in a storage area inc l udos transmitt i ng th e 
goneratod ono or more fi l es to the storage aroo through tho data l ink. 

4& — A m e thod for arch i v i ng a n i nt e ract i on by a user w i th a t e mn i na l d e v i c e 
hav i ng a d i sp l ay m e ans and a random acc e ss m e mory, th e m e thod 
compris i ng: 

uploading a fil e to a display dev i ce upon rece i v i ng a p l ayback 

r e qu e st, wh e r e in th e f ile i nciudes a r e pr e s e nt a tion of a s e ries of scr Q on 
d i sp l ays that a r e captur e d when th e us e r i nt e racts with th e t e rm i n al d e v i ce; 

display i ng th e s e r i es of scr ee n displays on disp l ay device 

accord i ng l y to a sp e cif ie d ord e r. 

47: The m e thod as r e c i t e d i n c l a i m 16, wh e r e th e sp e c i fi e d order is to 

d i sp l ay th e scre e n displ a ys s e quenti a l l y s o that progr e ss e s refl e cting how 
th e us e r int e racts w i th th e t e rminal d e v i c e can b e r e p l ayed. 

4&. The m e thod as r e c i t e d i n c l aim 17, wh e r e th e s e ri e s of scr ee n 

d is p l ays are respectfu ll y captur e d wh e n th e us e r e nt e rs a transaction 
through tho termina l dovico. 
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A^, — The m e thod as r e cit e d i n c l a i m 17, whoro tho ser i GS of screen 
disp l ays are respectful l y captured when th e user e nt e rs a n 
comm a nd to th e t e rm i na l d e v i c e . 

20^ — A syst e m for archiving an int e ract i on by a us e r with a t e rm i na l d e v i c e 
h a ving a disp l ay means and a random acc e ss m e mory, the syst e m 
compr i s i ng: 

a termina l dev i ce inc l ud i ng a m e mory spac e and a scr ee n, the 
m e mory s pac e i ncluding data for d i spl a y i ng on th e scr ee n, wher ei n th e 
t e rm i na l d e v i c e i s configur e d to g e n e rat e a s e ries of i mages from th e data 
when a user of the term i nal dev i c e i nt e racts th e rew i th; and 

a s e rv e r comput e r l ocat e d r e mote l y w i th resp e ct to th e t e rmin al 
d e v i c e and communicating with th e t e rmina l d e v i c e ov e r a data n e twork, tho 
s e rv e r including a storag e spac e a nd r e c ei v i ng on e or mor e f i les from th o 
t e rm i n al d e v i c e , the on e or mor e f ile s r e sp e ctiv el y r e pr e s e nting th e s e r ie s of 
i mages, whero i n tho server is conf i gur e d to archive the file in tho storage 
spac e . 

— The system of c l a i m 20, wh e r ei n th e term i n a l d e v i c e g e n e rates the 
on e or mor e files as soon as th e s e r ies of images are cached. 

23r. — Th e syst e m of claim 21 , wher ei n th e t e rm i na l d e vic e g e n e rat e s one 
or more attribut e s to b e assoc i ated w i th at l e ast one of th e on e or mor e 

I If WW* 
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— Tho syst e m of cla i m 20, whero i n th e t e rminal devic e is configur e d to 
caus e the one or more f i les to b e transmitt e d over the data n e twork to th e 
s e rv e r. 

24. — The syst e m of cla i m 20, where i n the son/or i nc l ud e s a s e rv e r modul e 
conf i gur e d to i nd e x the r e c ei v e d on e or more f i les i n th e storage spac e . 

2&Z — The syst e m of c lai m 2 4 , wh e r ei n th e server modu le i nclud e s an OCR 
e ng i n e to ana l yz e th e r e ce i v e d one or more f il es and automatically g e n e rat e 
k e ywords for ind e x i ng th e r e c ei v e d on e or mor e f ile s. 

2&: — Th e syst e m of cla i m 23, wh e r ei n th e server modul e inc l ud e s a 
proc e ss i ng e ng i n e to par se and s el ectiv el y process f il e compon e nts i n e ach 
of tho one or mor e fil e s. 

2^. Th e syst e m of c l a i m 20, wh e r ei n th e s e rv e r inc l ud e s a n int e rfac e 

conf i gured to r e tr i ev e th e on e or more f i les from the storag e spac e wh e n a 
p l ayback r e qu e st is r e ce i v e d. 

1 . A method for tracking predetermined activities for a terminal display, 
the method comprising: 

providing a series of displays on the terminal display, at least some 
of the displays requiring interactions from a user and being 
referred to as interactive displays: 
capturing one of the interactive displays after the one of the 

interactive displays has been altered with at least one interaction 
from the user in accordance with a predetermined reouirement: 
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saving the captured display into an image: 

continuing to successively display a next one of the interactive 
displays till a last one of the interactive displays, wherein each of 
the interactive displays is captured in a seouence of being 

displayed, and each of the captured displays includes at least 
one interaction from the user in accordance with a predetermined 
requirement; and 
sending at least some of the captured displays to a server. 

2. The method of claim 1 further comprising generating one or more 
attributes to be associated with each of the captured displays. 

3. The method of claim 2, wherein the one or more attributes includes 
an alphanumeric character string. 

4. The method as recited in claim 3, wherein the alphanumeric 
character string Is encrypted. 

5. The method as recited in claim 3. wherein the alphanumeric 
character string pertains to a time at which any of the interactive displays 
was altered. 

6. The method as recited in claim 2. wherein the one or more attributes 
include one or more of (1) a time stamp, (2) an electronic signature. (3) 
terminal device location information, (4) information relating to the terminal 
display. (5) user infonnation. (6) relative seouence index, or (7) system 
provided information. 



29 



12/06/03 



7. The method as recited in claim 1 . wherein the interaction includes 
one or more of (i) an entry bv the user, (ii) a click bv the user and (ill) a word 
or phrase. 

8. The method as recited in claim 1 , wherein the sending of at least 
some of the captured displays to the server includes compressing the 
captured displays into a file according to a compression scheme. 

9. A method for tracking predetermined actiyities for a terminal display, 
the method comprising: 

uploading a file to a display device upon receiying a playback 
reouest. wherein the file includes a representation of a series of 
captured screen displays of a terminal display associated with a 
user, wherein each of the screen displays reflects at least a 
change entered by the user in comparing with an original version 
thereof: and 

displaying the series of screen displays on the display device in a 
specified order to show how the user has altered each of the 
screen displays. 

10. The method as recited in claim 9, where the series of screen 
displays is analyzed bv an OCR. 

11. A terminal device for tracking predetermined activities therewith, the 
terminal device comprising: 

a display screen: 
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a memory space provided with data, the data configured to generate 
a series of displays for the display screen, at least some of the 
displays reouiring interactions from a user and being referred to 
as interactiye displays: 

an embedded module automatically triggered to capture a portion of 
the data corresponding to one of the interactive displays after the 
one of the interactive displays has been altered with at least one 
interaction from the user in accordance with a predetermined 
reguirement. wherein the embedded module is configured to save 
the portion of the data and fonA^ard a file including the portion of 
the data to a server. 

12. The terminal device of claim 1 1 . wherein the file includes other 
captured data related to some of the Interactive displays 

13. The terminal device of claim 12. wherein the embedded module 
includes generating one or more attributes to be associated with the portion 
of the data. 

14. The terminal device of claim 1 3. wherein the one or more attributes 
includes an alphanumeric character string. 

15. The terminal device of claim 14. wherein the alphanumeric character 
string is encrypted. 

16. The terminal device of claim 14. wherein the alphanumeric character 
string pertains to a time at which any of the interactive displays was altered. 
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17. The terminal device of claim 14. wherein the one or more attributes 
include one or more of (1) a time stamp, (2) an electronic signature, (3) 
terminal device location information, (4) information relating to the terminal 
display. (5) user information. (6) relative seouence index, or (7) system 
provided information. 

1 8. The terminal device of claim 1 1 , wherein the interaction includes one 
or more of (i) an entry bv the user, (ii) a click bv the user and (iii) a word or 
phrase. 

1 9. The terminal device of claim 1 1 . wherein the embedded module 
includes compressing the portion of the data according to a compression 
scheme. 

20. The terminal device of claim 1 1 , wherein the file pertains to image 
pixels and is subjects to be analyzed bv an OCR engine at the server 
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METHOD AND SYSTEM FOR TRACKING SCREEN ACTIVITIES ¥-FQR 
MULTI - SCREEN TRANSACT I ONS 



Techniques to facilitate a system to capture, process, and archive a 
series of user interactive events and subsequently retrieve the stored user 
interactive events are disclosed. The captured infomnation is indexed and 
stored for future access either on a terminal device or an accessible remote 
server device. 
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